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SYSTEMS AND METHODS FOR FACILITATING THE PEER REVIEW 

PROCESS 

FIELD OF THE INVENTION 

The present invention relates generally to systems and methods for conducting 
5 peer review. More particularly, the present invention relates to systems and methods 
for an automated peer review process. 

BACKGROUND OF THE INVENTION 

Peer review is a method used by Universities, Scholarly Journals, Government 
Agencies, Foundations and the like to review and evaluate the worthiness or value of 
10 papers submitted, for example as a part course work, or for publication, or as a 

proposal for a grant. Schools, and in particular primary and secondary schools, also 
g use peer review to provide feedback for improvement. Peer review is typically carried 

^1 out by several reviewers, to mitigate the effect of any prejudice which may influence 

jp& the opinion of a single reviewer. The reviewers typically analyze the papers for 

m 15 strengths and weaknesses, and typically provide a written end result, such as, for 
IP example, comments, a grade, a recommendation with respect to publication or funding, 

h and/or suggestions for improvement. Current methods for peer review suffer from 

problems of being too time consuming, wasteful in that a complete set of documents 
must typically be produced for each reviewer, and costly when such documents must 
20 be delivered to, and returned by, each reviewer by post or courier. In addition, it may 
not be possible when conducting peer review using manual means to completely 
obviate any prejudice through randomness or anonymity when desired, since a human 
is involved in manual methods of selecting and distributing papers to reviewers, and 
may either overtly or inadvertently communicate information regarding the authors to 
25 the reviewers. Moreover, in a school environment where handwritten papers are 

turned in, handwriting is frequently recognizable and identifiable as belonging to a 
particular individual, making a true "blind" review impossible. 
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What is needed are systems and methods for efficiently automating the process 
of peer review, while providing flexibility which has hitherto not been available 
through manual methods. 

SUMMARY OF THE INVENTION 

5 The present invention relates to automated systems and methods for conducting 

peer review. In one embodiment, the present invention provides a peer review system 
including a user interface for identifying the user, for accepting predefined user 
information, and for providing a result. 

There are typically three distinct kinds of users: sponsors, submitters, and 
10 reviewers. Sponsors are those who require or invite the submission of papers and 

define the criteria for the peer review. Submitters are those who create and submit the 
q papers to be reviewed. Reviewers are those who review the papers. Sometimes the 

Wj reviewers may also be the submitters or the sponsors. 

SI In the present invention, a peer review application is operably linked to the user 

U 15 interface and includes knowledge base information and defined rules for (1) accepting 
^ a paper for peer review, (2) defining the peer review assignment; (3) assigning the 

P paper to one or more of a defined set of reviewers for review, (4) providing to each 

K reviewer the criteria for reviewing each said paper to produce a peer review result, and 

(5) processing all peer review results for a paper to produce a peer review report for 
fU 20 that paper. A peer review application of the present invention is stored on a computer 
system having computer memory and a computer processor. An intermediary service 
provider is most preferably operably linked to said computer system, for displaying the 
user interface and the result to the user via, for example, the internet or an intranet. 

The criteria by which a paper is distributed for peer review preferably includes 
25 rules for randomly assigning said paper to any reviewer except the submitter, and for 

assigning to each reviewer only the number of papers predetermined by the sponsor. 
Moreover, the identification of the submitter of each paper can be controlled to 
provide a true, double-blind review in which the identity of the submitters are not 
disclosed to the reviewers. 
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In the present invention, the system includes as a part of the knowledge base 
information selectable reviewing and/or grading criteria to be used in evaluating a 
paper. The sponsor may choose from among the stored criteria, or may create new 
reviewing criteria. Where new reviewing criteria are created by the sponsor, the peer 
review application can supplement the knowledge base information by adding the new 
grading criteria. 

In some embodiments, the peer review application is stored on computer 
readable medium (e.g., DVDs, CDs, hard disk drives, magnetic tape and servers for 
streaming media over networks). In other embodiments, the peer review application is 
stored on computer memory or a computer memory device. 

In some embodiments, the computer system comprises computer memory or a 
computer memory device and a computer processor. In some embodiments, the 
computer memory (or computer memory device) and computer processor are part of 
the same computer. In other embodiments, the computer memory device or computer 
memory are located on one computer and the computer processor is located on a 
different computer. In some embodiments, the computer memory is connected to the 
computer processor through the Internet or World Wide Web. In some embodiments, 
the computer memory is on a computer readable medium (e.g., floppy disk, hard disk, 
compact disk, DVD, etc). In other embodiments, the computer memory (or computer 
memory device) and computer processor are connected via a local network or intranet. 

In some embodiments, "a processor" may in fact comprise multiple processors 
in communication with each other for carrying out the various processing tasks 
required to reach the desired end result. For example, the computer of an intermediary 
service provider may perform some processing and the computer of a customer linked 
to the intermediary service provider may perform other processing. 

In some embodiments, the computer system further comprises computer 
readable medium with the peer review application stored thereon. In further 
embodiments, the computer system comprises the computer memory, computer 
processor, and the peer review application is located on the computer memory, and the 
computer processor is able to read the peer review application from the computer 



memory (e.g., ROM or other computer memory) and perform a set of steps according 
to peer review application. In certain embodiments, the computer system may 
comprise a computer memory device, a computer processor, an interactive device (e.g., 
keyboard, mouse, voice recognition system), and a display system (e.g., monitor, 
5 speaker system, etc.). 

In yet another embodiment, the present invention provides a method of peer 
review including (1) providing a user interface capable of receiving user information, 
including information for identifying the user; (2) providing a peer review application 
linked to the user interface, and including knowledge base information and defined 
10 rules for (a) accepting a paper for peer review, (b) defining a peer review assignment; 
(c) assigning the paper to one or more of a defined set of reviewers for review, (d) 
providing criteria to the reviewers for reviewing each said paper to produce a peer 
p review result, and (e) processing all peer review results for any paper to produce a 

H p eer review report; (3) providing a computer system for operating the peer review 

/= ! 15 application, wherein the computer system includes computer memory and a computer 

U processor, (4) providing a hosted electronic environment operably linked to the 

j~ computer system; (5) displaying the user interface on the hosted electromc 

P environment; (6) receiving user information by way of the user interface; and (7) 

■yj 

fj processing the user information with the peer review application to generate a peer 

EJ 20 review report for each paper submitted for review. 

fU In some embodiments of the system and methods of the present invention, the 

user interface is a written document capable of being viewed by a user. In further 
embodiments, the user interface is telephone, modem, or other electronic device 
capable of receiving responses from a user (e.g., responsive to pre-recorded telephone 
25 message of questions or questions presented by an operator). In preferred 

embodiments, the user interface is a graphical user interface (e.g., a user interface 
screen presented on a computer monitor). 

In some embodiments of the methods of the present invention, the user 
information is received by way of the user interface. While it would be possible to 
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receive user information by receiving oral communications, or by receiving a written 
document from user, in the preferred embodiments, the receipt of the user information 
is by way of electronic communication {e.g., over telephone lines, cable lines, or a 
broadcast electronic communication), and most preferably by information entered into 
a web site. 

In some embodiments of the methods of the present invention, user information 
is processed with the peer review application to generate a peer review report. In 
some embodiments, the peer review application is operably linked to the computer 
processor such that the peer review application is able to process the user information. 
In some embodiments, the peer review application is physically located in the same 
computer as the computer processor. In other embodiments, the peer review 
application is in a different computer than the computer processor and the peer review 
application and computer processor are operably linked {e.g., there is an electronic 
connection between the computer processor and the peer review application). In some 
embodiments, the electronic connection is selected from phone lines, cable lines, 
broadcast transmission, or combinations thereof. 

In certain embodiments, the user information provided by sponsors identify the 
sponsor and allow the system to verify the user as a sponsor for access purposes. 
Sponsor user information can also comprise or define, for example, information 
identifying users having access to their site, information identifying a set of submitters 
and/or a set of reviewers, information defining the parameters of a peer review 
assignment, such as, for example, last date for submission of papers, last date for 
completion of the peer review assignment, the criteria for reviewing papers, the 
method for assignment of papers to reviewers (random allocation, manual assignment, 
reviewer choice, or a combination thereof). 

In certain embodiments, the user information provided by submitters identify 
the submitter, allowing access to information provided by the sponsor such as, for 
example, information relating to the submission of papers. Papers submitted are 
provided with identification indicia which link the paper to the reviewer for purposes, 
among others, of creating and distributing the peer review report. 



In certain embodiments, the user information provided by reviewers identify 
them as reviewers, allowing access to information provided by the sponsor, such as, 
for example, information regarding the selection or assignment of papers to be 
reviewed and the criteria to be used in reviewing each paper assigned for review, and 
the date by which the peer review assignment is to be completed. 

In the preferred embodiment of the present invention, the peer review report 
combines the peer review results for each submitted paper into a single document. 
Preferably, the peer review report is displayed on a computer screen. Alternatively, 
the results can be displayed on paper. In particularly preferred embodiments, the 
results are displayed on a web site. 

In certain embodiments, the intermediary service provider comprises a hosted 
electronic environment. In some embodiments, the hosted electronic environment is 
located on the Internet. In other embodiments, the hosted electronic environment is 
located on the world wide web. In still other embodiments, the hosted electronic 
environment is located on an intranet. In preferred embodiments, the hosted electronic 
environment comprises a web site. 

DESCRIPTION OF THE FIGURES 

Figure 1 illustrates a preferred embodiment of a system of the present 
invention; 

Figure 2a illustrates the process followed by a sponsor when using a peer 
review application of the present invention; 

Figure 2b illustrates the process followed by a submitter when using a peer 
review application of the present invention; 

Figure 2c illustrates the process followed by a reviewer when using a peer 
review application of the present invention; 

Figure 3 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 

Figure 4 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 



Figure 5 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 

Figure 6 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 

Figure 7 shows a user interface screen provided in one embodiment of the peer 
review spplication of the present invention. 

Figure 8 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 

Figure 9 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 

Figure 10 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 11 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 12 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 13 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 14 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 10 shows a user interface screen provided in one embodiment of the peer 
review application of the present invention. 

Figure 15 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 16 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 17 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 18 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 



Figure 19 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 20 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 21 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 22 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 23a shows a first page of a user interface screen provided in one 
embodiment of the peer review application of the present invention; 

Figure 23b shows a second page of a user interface screen provided in one 
embodiment of the peer review application of the present invention; 

Figure 24 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 25 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 26 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 27 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 28 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 29 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 30 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 31 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 32 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 



Figure 33 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 34 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 35 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 36 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 37 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 38 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 39 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 40 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

Figure 41 shows a user interface screen provided in one embodiment of the 
peer review application of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

The present invention relates to systems and methods for performing peer 
review. For example, the present invention provides systems, methods, and software 
tools for automatically generating peer review reports based upon predetermined 
criteria defined by the person or entity seeking the review. Most preferably, the peer 
review systems and methods of the present invention are integrated into a broader 
system for managing projects, academic environments and the like. 

To facilitate an understanding of the present invention, a number of terms and 
phrases are defined below: 

As used herein, the term "intermediary service provider" refers to an agent 
providing a forum for users to interact with each other (e.g., identify each other, make 



and receive assignments, etc). For example, an intermediary service provider may 
provide a forum for faculty members to create and distribute assignments to students in 
a class (e.g., by defining the assignment and setting dates for completion), or provide a 
forum for students to receive and respond to assignments such as peer review 
assignments. The intermediary service provider also allows, for example, users to 
maintain a portfolio of work submitted in response to all assignments for a particular 
class or project and for the collection of data (such as customized questions and 
rubrics) which can be used to supplement knowledge base data in a library of such 
data. In some embodiments, the intermediary service provider is a hosted electronic 
environment located on the Internet or World Wide Web. 

As used herein, the term "link" refers to a navigational link from one document 
to another, or from one portion (or component) of a document to another. Typically, a 
link is displayed as a highlighted or underlined word or phrase, or as an icon, that can 
be selected by clicking on it using a mouse to move to the associated page, document 
or documented portion. 

As used herein, the term "Internet" refers to a collection of interconnected 
(public and/or private) networks that are linked together by a set of standard protocols 
(such as TCP/IP and HTTP) to form a global, distributed network. While this term is 
intended to refer to what is now commonly known as the Internet, it is also intended 
to encompass variations which may be made in the future, including changes and 
additions to existing standard protocols. 

As used herein, the terms "World Wide Web" or "Web" refer generally to both 
(i) a distributed collection of interlinked, user-viewable hypertext documents 
(commonly referred to as Web documents or Web pages) that are accessible via the 
Internet, and (ii) the client and server software components which provide user access 
to such documents using standardized Internet protocols. Currently, the primary 
standard protocol for allowing applications to locate and acquire Web documents is 
HTTP, and the Web pages are encoded using HTML. However, the terms "Web" and 
"World Wide Web" are intended to encompass future markup languages and transport 
protocols which may be used in place of (or in addition to) HTML and HTTP. 
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As used herein, the term "Web Site" refers to a computer system that serves 
informational content over a network using the standard protocols of the World Wide 
Web. Typically, a Web site corresponds to a particular Internet domain name, such as 
"proveit.net/" and includes the content associated with a particular organization. As 
used herein, the term is generally intended to encompass both (i) the 
hardware/software server components that serve the informational content over the 
network, and (ii) the "back end" hardware/software components, including any 
non-standard or specialized components, that interact with the server components to 
perform services for Web site users. 

As used herein, the term "client-server" refers to a model of interaction in a 
distributed system in which a program at one site sends a request to a program at 
another site and waits for a response. The requesting program is called the "client," 
and the program which responds to the request is called the "server." In the context of 
the World Wide Web (discussed below), the client is a "Web browser" (or simply 
"browser") which runs on a computer of a user; the program which responds to 
browser requests by serving Web pages is commonly referred to as a "Web server." 

As used herein, the term "HTML" refers to HyperText Markup Language 
which is a standard coding convention and set of codes for attaching presentation and 
linking attributes to informational content within documents. During a document 
authoring stage, the HTML codes (referred to as "tags") are embedded within the 
informational content of the document. When the Web document (or HTML 
document) is subsequently transferred from a Web server to a browser, the codes are 
interpreted by the browser and used to parse and display the document. Additionally 
in specifying how the Web browser is to display the document, HTML tags can be 
used to create links to other Web documents (commonly referred to as "hyperlinks"). 

As used herein, the term "HTTP" refers to HyperText Transport Protocol which 
is the standard World Wide Web client-server protocol used for the exchange of 
information (such as HTML documents, and client requests for such documents) 
between a browser and a Web server. HTTP includes a number of different types of 
messages which can be sent from the client to the server to request different types of 
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server actions. For example, a "GET" message, which has the format GET, causes the 
server to return the document or file located at the specified URL. 

As used herein, the terms "computer memory" and "computer memory device" 
refer to any storage media readable by a computer processor. Examples of computer 
memory include, but are not limited to, RAM, ROM, computer chips, digital video 
disc (DVDs), compact discs (CDs), hard disk drives (HDD), and magnetic tape. 

As used herein, the term "computer readable medium" refers to any device or 
system for storing and providing information (e.g., data and instructions) to a computer 
processor. Examples of computer readable media include, but are not limited to, 
DVDs, CDs, hard disk drives, magnetic tape and servers for streaming media over 
networks. 

As used herein, the terms "computer processor" and "central processing unit" or 
"CPU" and "processor" are used interchangeably and refers to one or more devices that 
is/are able to read a program from a computer memory (e.g., ROM, RAM or other 
computer memory) and perform a set of steps according to the program. 

As used herein, the term "hosted electronic environment" refers to an electronic 
communication network accessible by computer for transferring information. One 
example includes, but is not limited to, a web site located on the world wide web. 

As shown in Figure 1, the preferred system of the present invention includes a 
user interface 10 operably connected to a computer processor 14 in communication 
with computer memory 16. Computer memory 16 can be used to store a peer review 
application 16a of the present invention, along with a central data base including 
papers submitted for review 16b, data for identifying subscribers 16c and other data 
and applications 16d. Most preferably, access to the user interface 10 is controlled 
through an intermediary service provider 12, such as, for example, a website offering a 
secure connection following entry of confidential identification indicia, such as a user 
ID and password, which can be checked against the list of subscribers 16c stored in 
memory. Upon confirmation, the user is given access to the site. Alternatively, the 
user could provide user information to sign into a server which is owned by the 
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customer and, upon verification of the user by the customer server, the user can be 
linked to the user interface 10. 

User interface 10 can be used by a variety of users to perform different 
functions, depending upon the type of user. For purposes of the present invention, 
there are preferably at least three categories of users (although other users may also be 
defined and given access): sponsors 18, submitters 20, and reviewers 22. Sponsors 18 
are those who require or invite the submission of papers, and define the parameters of 
those papers, including content. In an academic environment, this category typically 
includes teachers or professors. Submitters 20 are those who prepare and submit 
papers for review. In an academic environment, this typically includes students. 
Reviewers 22 are those who review the submitted papers for quality, and for 
compliance with the parameters and criteria defined by the sponsor. In an academic 
environment, reviewers can be the teacher or professor of the class for which the paper 
was submitted, other teachers or professors (e.g., members of a thesis or dissertation 
committee), or students. Indeed, the practice of having students exchange and grade 
tests and quizzes in class has been a common practice. While the preferred 
embodiment of the present invention is carried out in an academic setting, one skilled 
in the art will recognize that the present invention can also be applied to a variety of 
other peer review situations, such as, for example, evaluating papers for publication, 
and reviewing grant proposals. 

As shown in Figures 1-3, users preferably access the user interface 10 by using 
a remote computer, internet appliance, or other electronic device with access to the 
internet and capable of linking to an intermediary service provider 12 operating a 
designated website (such as, for example, turnitin.com) and logging in. Alternatively, 
if elements of the system are located on site at a customer's location or as part of a 
customer intranet, the user can access the interface by using any device connected to 
the customer server and capable of interacting with the customer server or intranet to 
provide and receive information. 

The user provides predetermined identification information (as shown in Figure 
3, this can include user type, email address, and password) which is then verified by 
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checking a "central database" containing the names of all authorized users stored in 
computer memory 16. If the user is not found in the central database, access is not 
provided unless the "free trial" option has been selected, and then access is only 
provided to sample screens to enable the unknown user to evaluate the usefulness of 
the system. The central database containing the identification information of 
authorized users could be maintained by the intermediary service provider or by a 
customer. If the user is known (i.e., contained within the list of authorized users), the 
user will then be given access to an appropriate "home page" based on the type of user 
and the user ID which links to subscription information and preferences previously 
selected by the user. Thus, "home pages" with relevant information can be created for 
sponsors, submitters, and reviewers. 

The login screen shown in Figure 3 allows the user to select the type of user 
interface to be accessed. Such a choice is convenient where an individual user fits into 
more than one category of user. For example, where an individual user is both a 
faculty member and a student in a class, allowing the individual to choose the user 
type will bring up the appropriate interface screen. In situations where there can be no 
overlap, such a choice, while preferable, will not be necessary since the central 
database can include each individual user's user type and can automatically bring up 
the appropriate user interface screen when the user signs in and is recognized. The 
user may also be given the option of selecting a secure session. 

Use of the System bv Sponsors 

As shown in Figures 1, 2a and 4, a sponsor accesses the user interface and 
logs in to the system to call up the sponsor's homepage. The sponsor's homepage will 
list all classes, projects or accounts being tracked for the sponsor. In the embodiment 
shown in Figure 4, the sponsor is a teacher tracking classes at three institutions. By 
selecting a particular class, the sponsor can access the records for that class. Using 
this screen, the sponsor can add classes or projects by clicking on the "add class" icon 
to the right of the institution name, or archive classes by clicking on the "A" icon to 
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the left of the class name. To check on the records for a specific class, the sponsor 
can click on the name of the class. 

As shown in Figure 5, a variety of class records can be maintained and 
accessed automatically. A class page navigation bar at the top of the page contains 
links which allow the sponsor to view a variety of records: "Inbox" can contain 
originality reports for papers turned in for the class; "Students" can contain a list of 
students in the class and links to their records; "Assignments" can contain a list of 
assignments for the class; "Reviews" can contain the peer review assignments for the 
class; "Calendar" can contain the due dates and post dates for assignments and peer 
reviews, holidays etc.; "Class Notes" can be used to post class notes; "Preferences" can 
be used by the Sponsor to set parameters for use of the system. Throughout the 
system, where the file contains more than one page, the page being viewed and all 
pages in the file can be shown, for example, at the bottom of the page, e.g.: page: 
[1] 2. The bracketed number is the page being viewed; the next page can be called up 
by clicking on the next number. In addition, other general information regarding use 
of the system can be accessed by clicking on links at the bottom of the page. Such 
general information can include the agreement regarding usage of the system, privacy 
obligations, instruction manuals for using the system, a tour of the system for first- 
time users, and/or a tutorial. Although these links are not shown in the remaining 
Figures, they preferably appear at the bottom of every screen when the system is in 
use. 

Figure 5 shows a class inbox, which can contain all submissions made to that 
class by each of the enrolled students, and can identify each assignment by student 
name, date submitted, and title. Icons provide links to the full text paper (under 
column "P") and to any originality reports which have been generated to check for 
plagiarism (under the column "R"). The sponsor is also given the ability to archive 
submitted work by checking the block to the far left of each submission. This would 
be useful for archiving the work of a submitter/student who has withdrawn before 
completion of the project/class or for archiving old work. 
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To view a portfolio of any specific student's work, the sponsor can click on the 
student's name. As shown in Figure 6, this produces a complete history of the 
student's submissions for the class. The sponsor can view any of the submissions by 
clicking on the appropriate icon. For example, the sponsor can view the originality 
report for the paper entitled "test 2" by clicking on the icon under the column "R" on 
the first line. The sponsor can read the full text of the paper entitled "test 2" by 
clicking on the icon under the column "P" on the first line, or by clicking on the title 
"test 2". The sponsor can review the two peer reviews of this student's paper entitled 
"A Test" by clicking on the "read" icon under the far right column "reviews." The 
sponsor can read the peer review submitted by this student on 26 January 2002 for 
another student's paper entitled "sample" by clicking on the icon under the column 
"PR" on the last line. 

As shown in Figure 7, the originality report for the paper "Whale Camp" 
shown in Figure 6 can be reviewed by clicking on the icon in column "R" to the left 
of the title "Whale Camp." When a student paper is submitted in response to an 
assignment, preferably by uploading it to the central database, originality can be 
determined by performing an originality analysis. The sponsor can initiate this process 
by selecting the account navigation bar icon "turn it in!" and selecting papers which 
have been submitted for originality analysis. 

Originality analysis is a process which typically consists of producing a digital 
fingerprint for the paper, and comparing the paper's digital fingerprint to the digital 
fingerprints of termpapers and documents stored in a database or gathered from the 
internet. Documents having digital fingerprints identified as a close match are then 
preferably compared full-text to the full-text paper to determine the level of 
duplication. An originality report, shown in Figure 7, can be created which includes a 
graphical indication of the likelihood of originality ("overall similarity index" ranking 
originality from 1 (least similar) to 5 (most similar)) and provides links to documents 
which contain matching passages, to enable the sponsor to view the flagged passages 
and make a judgment on whether plagiarism has occurred. In addition, textual 
passages in the paper for which matches were found can be identified. 
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In the preferred embodiment, the steps of the process are carried out by the 
intermediary service provider, and the report is generated and accessible to the sponsor 
through the user interface. However, some institutions may wish to maintain control 
over their student's papers. In such cases, it is possible to divide the processing 
between the customer's server and the intermediary service provider's server. For 
example, the papers may be uploaded and stored in the customer's database, and the 
customer's processor will create a fingerprint of the paper. The fingerprint can be 
checked by the customer's processor against the fingerprints of other papers stored on 
the customer's database. Then, the fingerprint of the paper can be transmitted to the 
intermediary service provider for processing (e.g., comparison with the other 
documents stored by the intermediary service provider). Either the intermediary 
service provider server can then do the final, full-text comparison to produce the 
originality report, or the intermediary service provider server will transmit to the 
customer server the information regarding the documents which were identified as 
potential "hits" during the comparison, so that the customer server can produce the 
final originality report. 

The "assignments" account navigation bar icon provides access to the 
assignments page, an example of which is shown in Figure 8. This page shows all 
assignments for the class, including start date, due or end date, a "post" date (when 
students may be given access to peer reviews and/or grades for the assignment), and a 
title. A reminder date may also be selectable, whereby a reminder (for example, by 
email) can be sent to the submitter to remind the submitter of the due date for the 
assignment. The sponsor can update the assignment by selecting the "U" icon or 
delete the assignment by selecting the trashcan icon. In the preferred embodiment, two 
kinds of assignments can be created: a new paper assignment or a new peer review 
assignment. To create a new paper assignment, the sponsor clicks on the new paper 
assignment icon to access an assignment screen identifying the title, description, and 
instructions for completing and uploading the new paper assignment. 

In the preferred embodiment, the sponsor can select a complete peer review 
assignment from a library of complete peer review assignments, or can create a new 
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peer review assignment using a five step process is used to define the peer review 
assignment. To create a new peer review assignment, the sponsor selects the "create a 
new peer review assignment" icon to access the screen shown in Figure 9. 

In the first step, a title for the peer review assignment is provided by the 
5 sponsor along with any description and/or additional instructions desired by the 
sponsor. The sponsor then selects the "next" icon to go to step 2. 

As shown in Figure 10, the criteria for the peer review assignment can be 
established by the sponsor. These criteria preferably include (1) identity of the paper 
assignment this peer review is to be paired with; (2) relevant dates such as, for 
10 example, a start date, a due date (e.g., the date by which the peer review must be 
completed and uploaded), a post date (e.g., the date when the results of the peer 
review will be available to interested parties), and possibly a reminder date (e.g., the 
date on which a reminder will be sent to the reviewers to remind them of the 
jj upcoming due date for the completion of the peer review assignment); (3) the method 

15 by which the papers will be distributed to the students/reviewers; (4) dissemination of 
fc* ratings for the reviewed papers; and (5) keywords related to the assignment to enable 

the sponsor to access and review relevant topical questions to be answered by the 
reviewers stored in the central data base. 

The sponsor is preferably able to change the assignment if necessary before the 
20 "start" date. The sponsor can also, if desired, select a "post date" which occurs after 
the due date to provide adequate time for the sponsor to check all reviews and make 
any adjustments to grades which might be warranted under the circumstances. 

The method by which papers will be distributed to the students/reviewers is 
also preferably selectable to allow the sponsor to determine whether papers will be 
25 distributed to individuals or to groups. Where distribution is to occur to individuals, 

the sponsor will preferably be able to determine how many papers each student will 
review and to choose random or manual distribution of papers. Where distribution 
will occur to groups, the sponsor will identify the groups and then determine the 
method by which papers will be distributed to each group (e.g., manually, randomly, 
30 or by exchange between groups). 
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Before the peer review assignment is created, and before distribution occurs, 
the sponsor may wish to review each paper submitted to make certain that personally 
identifiable information is not included in the body of the paper. Assuming anonymity 
is desired, and any such personally identifying information is removed, the method of 
distribution can be determined. 

For example, as shown in Figure 10, the sponsor has chosen to have each 
student review two papers, and has selected one paper to be randomly assigned to each 
student, and to allow manual assignment of one paper to each student. Random 
assignment will most preferably distribute a paper randomly to the universe of students 
who are not the author. Likewise, manual selection will preferably be controlled to 
prevent review of a paper by its author and to remove manually selected papers from 
the universe of papers available for review to insure that all papers receive neither 
more nor less than the desired number of reviews. This can be done by allowing 
students to select any paper other than their own and papers already selected by others, 
or by allowing the faculty member to manually assign papers to students. 

The sponsor can also determine whether or not a grade will be given and/or 
who will have access to the grade received by any paper. The choices provided by 
Figure 10 include "hide grade" (the grade is only known to the sponsor and is not 
disclosed to submitters, reviewers or others), "show to author" (the grade is only 
transmitted to the paper's author), or "show to all" (the grade for each paper is 
disclosed to all authorized users). 

Finally keywords can be provided to enable the sponsor to access questions and 
rubrics stored in the library. By selecting the "custom" icon (to create custom topics) 
or the "library" icon (to select stored topics) at the bottom of Figure 10, the sponsor 
moves to step 3. 

As shown in Figure 11, the sponsor can select or create criteria, such as topical 
questions to be answered by the reviewer, and the minimum length, if any, for the 
response. The topic question can be created by the sponsor or selected from one or 
more libraries of topic questions (an example of which is shown in Figure 12). The 
system most preferably allows sponsors to add questions to a library. For example, the 
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sponsor may wish to add standard questions used in the past by the sponsor, or 
questions recommended by a textbook publisher, or state or district educational 
authority. The sponsor preferably is given the choice to share such questions or 
rubrics with other sponsors. 

Stored topic questions can be conveniently categorized into sublibraries directed 
to such areas as thesis/introduction, organization, style, grammar/mechanics, evidence, 
conclusion, and general, with each sublibrary accessible by selecting the appropriate 
icon. When a desirable topic question is located, it can be used in the assignment by 
selecting or clicking on the "check" icon to the right of the question to be added. 
When the sponsor creates a new topic questions, the library is preferably supplemented 
by adding the new topic questions. 

When acceptable topic questions have been created or selected, the sponsor 
selects the "next" icon at the bottom of the page to move to step 4. 

As shown in Figure 13, the sponsor can establish yet other criteria in the form 
of rubrics for rating selected aspects of the paper. A rubric is a question which asks 
the reviewer to rate an aspect of the paper on a defined scale, for example: "From 0 
to 5 rate the student's effectiveness in identifying the principal leadership 
characteristics of Napoleon Bonaparte." Preferably a library of stored rubrics is 
accessible to the sponsor by selecting the rubric library icon. Where a sponsor creates 
a new rubric, the library is preferably supplemented by adding the newly-created 
rubric. Once all rubrics have been selected, the sponsor selects the "next" icon to 
advance to the final step. 

As shown in Figure 14, the final step allows the sponsor to review all the 
criteria for the peer review assignment, and to make any changes needed, before 
selecting the "submit" icon to create the peer review assignment. 

As shown in Figure 15, once the "submit" icon is selected, the Assignments 
page shown in Figure 8 is updated, for example by adding the newest assignment to 
the bottom of the list. Alternatively, it would also be possible to update or supplement 
the assignment page by adding the newest assignment to the top of the list of 
assignments, or by sorting alphabetically, by end date, by start date, or by any other 
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sortable criteria. This screen also allows the sponsor to create a manual paper 
exchange for peer review purposes, by selecting the pencil icon under the column 
marked "exchange". When this icon is clicked, the "exchange" screen shown in 
Figure 16 is accessed. 

Using the "exchange" screen of Figure 16, the sponsor can manually assign for 
review specific papers to specific students. The sponsor needs only select a paper then 
click the update icon next to the name of the student he wishes to review the selected 
paper. The number of the paper then appears in the "reviewing" box next to the 
student's name. The "x" appearing in the box identifies a random paper assignment 
to be made by the system. In the event of manual assignment, the system for 
randomly assigning papers would eliminate both the manually assigned paper, as well 
as any papers authored by the reviewer, from the universe of papers to be randomly 
assigned, to prevent possible duplication (i.e., a reviewer being assigned the same 
paper twice, or a paper authored by the reviewer). In the preferred embodiment, 
manual assignment of papers to review through the "exchange" screen takes place prior 
to the "start" date selected for the assignment. Once an assignment is made, the 
information identifying the assignment is preferably posted to a central class or project 
calendar (Figure 17) accessible to all relevant users. The central calendar can also be 
used to provide other information or links such as, for example, scheduling 
information, holidays, office hours, lecture notes, examinations, tests and quizzes, 
announcements, and the like. For convenience, this page is preferably accessible from 
other pages in the sponsor/faculty user class interface by selecting the "calendar" class 
account navigation bar icon. 

The status of the peer review assignments can be viewed by selecting the "peer 
review" account navigation bar icon to access the page shown as Figure 18. This 
page allows the sponsor to read student papers, view peer review summary statistics 
and grades, and to read the reviews of the papers. This page preferably identifies each 
paper to be reviewed and the author of each paper, along with the due date and 
posting date. When a review is posted/uploaded, the number of reviews posted to date 
is shown for each paper, as is the score or average score if more than one review has 
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been posted. When all reviews have been posted, a grade is also assigned based on 
predetermined criteria. An icon is preferably activated when a review is posted which 
permits the sponsor to read all reviews which have been posted. In addition, the 
sponsor may also create a review of the paper by selecting the pencil icon in the "post 
review" column. 

When the sponsor wishes to review a selected paper, the pencil icon in the 
"post review" column of the page shown in Figure 18 is clicked on to access the page 
shown in Figure 19. This page can provide questions and rubrics which are identical 
to those being used by the other reviewers, or it can be customized to provide other 
questions and rubrics. In addition, short adjectives or phrases, such as, for example, 
"thoughtful" "concise" "incomplete" "disorganized" etc. can be provided by the 
sponsor to describe his or her overall impression of the paper. This field could also, if 
desired, be provided to the other reviewers. An optional field is also preferably 
provided which enables the sponsor to enter a grade for the paper. Upon completion 
of the review, the sponsor selects the "submit" icon at the bottom of Figure 19 to 
update and return to the screen shown in Figure 18. 

The sponsor can read the submitted reviews by clicking the icon in the "read" 
column of Figure 18. This accesses a peer review page (Figure 20) which shows 
relevant summary information relating to all reviews such as, for example, the average 
score by rubric, reviews which have been posted, the individual score by each 
reviewer, comments by each reviewer, the identity of each reviewer, and a link to the 
full text of each review showing the responses to the topical questions and rubrics 
(Figure 21). The full text of each review, shown in Figure 21, also preferably 
provides a link (shown at the top of the page) to enable the sponsor to read the paper, 
as well as the option of hiding the review, if desired, so that it is not disclosed to the 
students. 

The system also preferably allows sponsors to establish their preferences by 
selecting the account navigation bar icon marked "preferences." This provides access 
to the screen shown in Figure 22, which preferably allows global preferences for the 
user interface, such as, for example, the color of the command bar, the homepage 
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name and address of the sponsor, the number of items to be displayed on a page, 
whether detailed page descriptions should be shown, etc., to be selected or changed. 
Additionally, preferences for each class or project can also be provided, such as, for 
example, the name and address for each class homepage, others who can view the 
work of submitters, what documents will be accessible to the submitters, etc. 

Finally, a "help!" icon is preferably provided on the system navigation bar 
which provides information to help the user navigate the system. An example of a 
typical "help!" screen is shown at Figures 23a and 23b. Pop up help screens are also 
used throughout the system where appropriate. For example, the first time a sponsor 
wishes to create a peer review assignment, a screen can pop up to ask the sponsor if 
he or she wishes to review the tutorial 

Use of the System bv Submitters 

As shown in Figures 1, 2b and 24, a submitter, such as, for example, a 
student, accesses the user interface and logs in to the system to call up the submitter's 
homepage. The submitter's homepage will list all classes, projects or accounts being 
tracked for the submitter. In the embodiment shown in Figure 24, the submitter is a 
student enrolled in two classes. Optionally, by selecting the "join new class" icon on 
the Account Navigation Bar, the submitter can add new classes to the home page and 
track all classes for which he enrolls or all projects in which he is a participant. By 
selecting a particular class (e.g., by clicking on the name of the class), the submitter 
can access their Class (or Project) portfolio. The portfolio, shown in Figure 25, 
contains a list of all assignments submitted during the class. The class portfolio, and 
all other pages in the class account, contains a Class Account Navigation Bar across 
the top which provides icons for navigating the system, including "class portfolio," 
"assignments," "turn it in!," "peer review," "calendar," "class notes," and "preferences." 
The class portfolio preferably includes information regarding the type of assignment 
(paper, review, test, etc.), the date the work was submitted, the title of the work, and 
whether any reviews of papers have been posted. If reviews authorized for release to 
the submitter have been posted in the "reviews" column, the submitter can click on the 
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icon "read" to call up the reviews. A link may also be provided to enable the 
submitter to send a message (for example, via email) to the sponsor. 

As shown in Figure 26, if the sponsor allows the submitters to review the 
originality reports generated for papers, a link will be displayed under column "R" in 
the Class Portfolio. Clicking on an originality report icon will display the 
corresponding originality report, such as that shown in Figure 7. 

The submitter can review upcoming assignments by selecting the "assignments" 
icon from the class account navigation bar. The assignments page, shown in Figure 
27, preferably displays the type of assignment (paper, peer review, etc.), the start date, 
the due or end date, the date when results (such as grades, reviews, etc, will be 
posted), the title of the assignment, and the current status. Every time the submitter 
completes an assignment by uploading to the system, this page will be updated to 
show the status of the assignment as "complete." For partial submissions (for example, 
where a number of papers are to be reviewed), the status column will be updated to 
show the number completed. 

To determine the parameters and criteria for any assignment, the submitter 
clicks on an assignment title to go to a page, such as that shown in Figure 28, 
containing the detailed instructions for completing that assignment. 

To submit a paper, the submitter selects the "Turn it in!" icon on the class 
account navigation bar to access the paper submission page shown in Figure 29. This 
page allows the submitter to provide the paper title and the author's first and last name 
and ID, and select the assignment for which the paper is being submitted. The text of 
the paper, abstract, and bibliography is preferably "cut and pasted" into the places 
provided on this page. By using the "cut and paste" method it is possible to avoid 
problems typically encountered with attempting to upload papers saved in different 
formats. However, alternatively, it should be possible to save the paper in a specified 
format (such as, for example, Microsoft® Word, WordPerfect®, Rich Text Format) 
and provide a link for uploading the file to the system. Once the information has been 
provided, the submitter selects the "submit" icon at the bottom of the page to upload 
the paper to the system. 
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To access the central class calendar, the submitter selects the "calendar" icon on 
the class account navigation bar to access the calendar as shown in Figure 17. 
Assignments can be accessed from this page by clicking on any assignment shown on 
the calendar. 

Any posted class notes can be accessed by selecting the "class notes" icon on 
the class account navigation bar. The submitter's preferences can be set or modified 
by selecting the "preferences" icon on the class account navigation bar. 

Use of the System by Reviewers 

In an academic setting, the submitters/students may also be the reviewers. In 
that event, the peer review function is included with the student's class account as 
shown in Figure 2B. To submit a peer review, the student either clicks on the title of 
the peer review assignment of the page shown in Figure 27, or selects the "peer 
review" icon on the class account navigation bar to access the peer review page shown 
in Figure 30. This page provides information regarding when the review is due, 
including date and time, and when the reviews will be posted. A list of all class 
papers is provided, and a review icon (in this case, a pencil) is displayed next to the 
papers to be reviewed. By clicking on the review icon, the page shown in Figure 31 
is displayed. The student may choose to examine the topical questions and rubrics 
contained in Figure 31 first, and then go back to access and review the paper after 
ascertaining the standards for review. Once the paper has been examined, the 
student/reviewer can return to the peer review page, respond to the topical questions 
and rubrics, as shown in Figure 32, and complete the peer review assignment by 
selecting the "submit" icon at the bottom of the page shown in Figure 32. The 
reviewer can also mark up the paper on-line, with the changes being highlighted using 
any conventional method such as, for example, red-lining. 

In the event the student's/reviewer's response does not meet the criteria set by 
the sponsor (for example, the minimum length of a response to a topical question is 
not met), an error message can be generated and/or the submission not accepted until 
correction is made. An example of such an error message is shown in Figure 33 just 
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below the class account navigation bar. A similar error message could be generated if 
there are other faults such as, for example, the student's failure to rate the paper using 
one of the rubrics in Section C. 

As shown in Figure 34, if the student returns to the peer review page after the 
due date, the ability to review the assigned papers is preferably removed (for example 
by eliminating the icon in the "post review" column - compare Figure 34 with Figure 
30). If the student returns to the peer review page after the post date, and if the 
sponsor has elected to make such information available to students, information 
relating to the reviews will be displayed. This may include the number of reviews 
submitted for each paper, the actual reviews may be accessible by clicking an icon 
(under the title "read reviews"), marked-up copies of the papers may also be available 
along with the paper as originally written, and summary information may also be 
shown for each paper, such as, for example, the statistical graded average for the 
reviews, grades and the like. By clicking on the "read reviews" icon, students can 
access the page shown in Figure 35. This page shows the average scores for the 
selected rubrics, and a summary for each individual review, showing the date 
submitted, the score, the "comments" (entered as adjectives or short phrases in Section 
B of the review shown in Figure 33), and an icon for accessing the full review. By 
clicking on the "full review" icon, the student accesses the page shown at Figure 36 
which shows the full responses to the topical questions and rubrics which form the 
basis for the review. In addition, a link to the paper reviewed can be provided to 
allow the person reading the review to go to the paper. 

In situations where the reviewer is not also a submitter, a reviewer home page 
can be created, such as that shown in Figure 2C, which can be accessed and navigated 
in substantially the same way as the other user pages described above (e.g., by logging 
on and providing identification information). A list of projects could be provided on 
the reviewer home page which lead to a project page including any peer review 
assignments for specific projects. For example, a reviewer may be a scientist 
responsible for reviewing papers for publication in a journal and also reviewing grant 
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applications. These could be considered as two distinct projects which would appear 
on the reviewer home page. 

If the reviewer selects one account page, such as, for example, a journal 
account page, the navigation bar might include links to a central calendar providing 
publication deadlines for specific issues which drive the dates for reviewing papers to 
be published in those issues, as well as a portfolio showing reviews already submitted. 
As described above, the navigation bar could include a "peer review" icon which will 
lead to a peer review page identifying papers submitted for publication and indicating 
those papers to be reviewed by the reviewer. Once reviews are completed and 
submitted to the sponsor inbox, the portfolio and peer review page can be updated to 
show the completed action. Once the post date is passed, the reviewer can also review 
the peer reviews submitted by other reviewers for the same or other papers. A peer 
review for articles submitted for publication could well contain additional information, 
including a recommendation on whether or not to publish the article, and whether or 
not the author needs specific revisions to the work before publication should occur. 

If the reviewer selects a different account page, such as, for example, a grant 
program account, the navigation bar might include links to a central calendar 
providing, for example, dates for submitting materials for grants, dates for reviewing 
grant submissions, and dates for announcing the award of grants, etc. As described 
above, the account navigation bar could include a "peer review" icon which will lead 
to a peer review page identifying grant applications submitted for consideration, and 
icons which indicate which grant applications should be reviewed by the reviewer. As 
noted above, a sponsor will establish the topical questions and rubrics to be followed 
in evaluating the grant applications. In addition, the peer review page will likely also 
include a recommendation on whether or not the proposed work should be funded 
and/or the extent to which funding should be made. 

Regardless of the situation under which the review occurs, the identity of the 
reviewers, while known to the sponsor, is most preferably not disclosed to the 
submitters or other reviewers, since reviewer anonymity in peer review situations 
promotes candid, honest reviews. However, to provide maximum flexibility, the 
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system can be provided with the option of disclosing the reviewer's or submitter's 
identities. Moreover, the system can be set up to provide for more than one round of 
reviews. 

Use of the System By Other Users 

Users other than sponsors, submitters and reviewers may have access to the 
user interface. For example, an institution having more than one sponsor (such as a 
college with many professors, a journal with many reviewers and the like) may wish to 
appoint an account administrator, who can sign in and access the system as an Account 
Administrator. Figure 37 shows a typical Account Administrator home page which 
can provide information for each authorized sponsor/professor such as, for example, 
the user ID and name of each sponsor authorized to access the system using the 
institution's account. The Account Navigation Bar includes icons which enable the 
Account Administrator to add new sponsors/professors, to edit entries for existing 
sponsors/professors, to deactivate professors (for example, by checking the blank box 
to the left of the entry for that sponsor/professor). Deactivation by the Account 
Administrator will deactivate all classes for that sponsor/faculty member, and block 
further access by other users to class records for that sponsor/faculty member. By 
clicking on the name of the sponsor/faculty member, the Account Administrator can 
review the Class Statistics page for that sponsor/faculty member as shown in Figure 
38. 

The Class Statistics page for each sponsor/faculty member can include a list of 
each class enrolled in the system, along with the class ID for each. In addition, 
selected statistics for that sponsor/faculty member may also be provided, such as, for 
example, the total number of classes, number of students in those classes, total number 
of submissions, including total number of papers or reports, peer reviews, and digital 
portfolios. The page shown at Figure 38 may also include a function which enables 
the Account Administrator to deactivate any one or more of the classes/accounts 
shown. For example, if an account is created for a class in advance of the start of a 
semester, and the class is subsequently cancelled due to low enrollment, the Account 
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Administrator can deactivate the account established for that class by, for example, by 
clicking the box to the left of the class name. To examine the statistics for each listed 
class, the Account Administrator can click on the class name to access the page shown 
at Figure 39. 

Account administrators can add sponsors within their institution by providing 
each sponsor with the necessary account enrollment information, or they can manually 
add the sponsor, for example by clicking on the "add instructor" icon shown in Figure 
37 and providing the sponsor's email address after accessing the screen shown in 
Figure 40. 

Account administrators can also manage the preferences for their user profile 
and for their institutions account by selecting the "preferences" icon on the Account 
Navigation Bar and entering the information relating to preference selections on a 
screen like that shown at Figure 41. 

Yet other users may be authorized to access the system. For example, parents 
may be given access to their student's class calendars, assignment pages, and class 
portfolios. Visitors, such as other institutions, may be authorized to access the system 
on a free trial basis in order to evaluate the system for use at their institution. Such 
trial use would not permit such visitors to access accounts established by authorized 
users, but would permit the visitors to create a trial account, create assignments, 
submit papers, create and submit peer reviews, and perform all functions on a trial 
basis to verify the suitability of the system for use. 

The present invention is not limited by the nature of the user. The user may be 
an individual, institution or any other entity. Any user involved in peer review 
activities may find beneficial use for the integrated system, software and methods of 
the present invention. The description provided above illustrates some uses of the 
systems and methods of the present invention, and are specifically directed to the 
preferred embodiments of the invention, and are not meant to limit the scope of the 
present invention. Various modifications and variations of the described method and 
system of the invention will be apparent to those skilled in the art without departing 
from the scope and spirit of the invention. Although the invention has been described 

-29- 



in connection with specific preferred embodiments, it should be understood that the 
invention as claimed should not be unduly limited to such specific embodiments. 
Indeed, various modifications of the described modes for carrying out the invention 
which are obvious to those skilled in the relevant fields are intended to be within the 
scope of the following claims. 
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